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(54) Image-capturing apparatus and image processing apparatus 



(57) An innage-capturing apparatus, includes: an im- 
age-capturing unit that captures an image of a subject 
through an optical system ; and a control unit that outputs 
images captured at the image-capturing unit as image 
data files, and: the control unit outputs the image data 
files in a format that allows an image data file containing 
a normal image to be distinguished from an image data 
file containing a reference image to be used to correct 



FIG.3 



an image defect in the normal image. An image process- 
ing apparatus, includes: an input unitto which an image 
data file output from the image-capturing apparatus is 
input; an identifying unit that identifies the image data 
file as a file containing the reference image or a file con- 
taining the normal image; and a correction unit that cor- 
rects an image defect in the image identified as the nor- 
mal image based upon the image identified as the ref- 
erence image. 
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Description 

[0001 ] The present invention relates to an image-capturing apparatus sucli as a digital cannera and an image process- 
ing apparatus that processes an image captured by the image-capturing apparatus. More specifically, it relates to an 
5 Image-capturing apparatus and an image processing apparatus to be utilized in conjunction with each other so as to 
output captured image data from the image-capturing apparatus to the image processing apparatus and eliminate 
image defects such as any adverse effect of foreign matter in the image-capturing optical path from the captured image 
at the image processing apparatus. 

[0002] If foreign matter becomes adhered to an optical member such as an optical filter disposed in the image- 

10 capturing optical path near the image-capturing element in an image-capturing apparatus such as a digital camera, a 
shadow of the foreign matter adhered to the optical member appears in a captured image, causing an image defect. 
Foreign matter such as dirt or dust particles tends to enter an image-capturing apparatus in which interchangeable 
lenses are used particularly readily during a lens replacement. An image captured in such an image-capturing apparatus 
is thus more susceptible to a defect caused by foreign matter having entered the image-capturing apparatus and 

15 becoming adhered onto an optical member. This problem is addressed in an image correcting apparatus in the related 
art by correcting an image defect manifesting in a captured image based upon position information which is set to 
indicate the position of the image defect (Japanese Laid Open Patent Publication No. H11-103913). 
[0003] It is necessary to set the image defect position information in the image correcting apparatus disclosed in 
Japanese Laid Open Patent Publication No. H 11-103413. This image correcting apparatus detects the image defect 

20 position information by using a captured image of a uniform plane in, for instance, white or black. If the lens of the 
image-capturing apparatus is repeatedly exchanged without removing foreign matter having entered the image-cap- 
turing apparatus, more foreign matter is allowed to enter the image-capturing apparatus and the image defect becomes 
worse. For this reason, it is desirable to detect the image defect position information to be set in the image correcting 
apparatus by using an image captured with timing as close as possible to the timing with which the correction target 

25 image is captured. However, in the image correcting apparatus in Japanese Laid Open Patent Publication No. H 
11-103413, the image used to detect the image defect position information and the correction target image must be 
individually selected and specified, which is a time-consuming process. 

[0004] According to the 1st aspect of the invention, an image-capturing apparatus comprises: an image-capturing 
unit that captures an image of a subject through an optical system; and a control unit that outputs images captured at 
30 the image-capturing unit as image data files, and: the control unit outputs the image data files in a format that allows 
an image data file containing a normal image to be distinguished from an image data file containing a reference image 
to be used to correct an image defect in the normal image. 

[0005] According to the 2nd aspect of the invention, it is preferred that an image-capturing apparatus according to 
the 1st aspect of the invention further comprises: a mode setting unit that sets a reference image acquisition mode in 
35 which the reference image is to be obtained, and: the control unit designates an image captured by the image-capturing 
unit when the reference image acquisition mode is set as the reference image. 

[0006] According to the 3rd aspect of the invention, it is preferred that in an image-capturing apparatus according to 
the 1 st aspect of the invention, the control unit appends different extensions to the image data file containing the nomnal 
image and the image data file containing the reference image. 

40 [0007] According to the 4th aspect of the invention, it is preferred that in an image-capturing apparatus according to 
the 1st aspect of the invention: the image data files are each constituted of image data corresponding to pixels in the 
image and information related to the image; and the control unit stores information with which the image data file 
containing the nonnal image and the image data file containing the reference image can be distinguished from each 
other in the information related to the image. 

45 [0008] According to the 5th aspect of the invention, an image-capturing apparatus comprises: an image-capturing 
unit that captures a subject image input through an optical system; a mode switching unit that selects a first mode in 
which a reference image is captured by the image-capturing unit or a second mode in which a normal image is captured 
by the image-capturing unit; an output unit that outputs the reference image and the normal image captured by the 
image-capturing unit as image data files to an external apparatus; and an information appending unit that appends 

50 infomnation indicating whether an image data file output by the output unit contains the reference image or the nomnal 
image to the image data file. 

[0009] According to the 6th aspect of the invention, it is preferred that in an image-capturing apparatus according to 
the 5th aspect of the invention: the information appending unit appends metadata containing information indicating 
either the reference image or the normal image to the image data file; and the meta data do not alter the reference 
55 image or the normal image contained in the image data file. 

[0010] According to the 7th aspect of the invention, it is preferred that in an image-capturing apparatus according to 
the 5th aspect of the invention: the information appending unit appends the information indicating whether the image 
data file contains the reference image or the normal image to the image data file by partially modifying a file name 
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assigned to the image data file. 

[0011] According to the 8th aspect of the invention, an image processing apparatus comprises: an input unit to which 
an image data file output from an image-capturing apparatus according to any one of the 1st aspect through the 4th 
aspect of the invention is input; an identifying unit that identifies the image data file as a file containing the reference 
5 image or a file containing the normal image; and a correction unit that corrects an image defect in the image identified 
as the normal image based upon the image identified as the reference image. 

[0012] According to the 9th aspect of the invention, it is preferred that in an image processing apparatus according 
to the 8th aspect of the invention 8: the image data files are each constituted of image data corresponding to pixels in 
the image and information related to the image; and the correction unit selects a specific reference image to be used 

10 to correct an image defect in a given normal image based upon the information related to the image contained in the 
image data file of the reference image and the information related to the image contained in the image data file of the 
normal image. 

[0013] According to the 10th aspect of the invention, an image processing apparatus, comprises: an input unit to 
which an image data file output from an image-capturing apparatus according to any one of the 5th aspect through the 
15 7th aspect of the invention is input; an identifying unit that identifies the image data file as a file containing the reference 
image or a file containing the normal image based upon the information; and an elimination unit that eliminates an 
effect of foreign matter present on an image-capturing optical path at the image-capturing unit from the normal image 
based upon the reference image. 

[0014] According to the 1 1th aspect of the invention, a computer-readable computer program product comprises: a 
20 control program that executes a function of an image processing apparatus according to any one of the 8th aspect 
through the 10th aspect of the invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 

25 [0015] 

FIG. 1 shows the structure of an electronic camera in which interchangeable lenses are used. 

FIG. 2 shows a personal computer (PC) and its peripheral devices shown together with a block diagram of the 

electronic camera. 

30 FIG. 3 shows a flowchart of the processing executed in the electronic camera during a photographing operation. 

FIG. 4 shows the photographing procedure executed on the electronic camera side. 

FIG. 5 shows a flowchart of the processing executed at the PC when taking in image data. 

FIG. 6 shows a flowchart of the processing executed at the PC to eliminate the undesirable effect of foreign matter. 

FIGS. 7A and 7B show the local standardization processing executed on the brightness plane. 
35 FIG. 8 shows a histogram of the transmittance map. 

FIGS. 9A and 98 show the change in the foreign matter shadow position occurring as the pupil position changes. 

FIGS. 1 0A and 1 0B show the change in the size of the foreign matter shadow occurring as the aperture value, i. 

e., the F value, changes. 

FIG. 11 shows a table of the one-dimensional filter coefficients corresponding to the individual aperture values. 
40 FIG. 1 2 shows the filter used to convert the transmittance map to a transmittance map corresponding to the aperture 

value F16, represented as a two-dimensional filter. 

FIG. 13 shows the F value conversion executed to convert the transmittance over an area where foreign matter 
of medium-size is present. 

FIG. 14 shows how the program is provided in a recording medium such as a CD ROM or through a data signal 
45 on the internet or the like. 

DESCRIPTION OF PREFERRED EMBODIMENTS 

(Structures of Electronic Camera and Personal Computer) 

50 

[0016] FIG. 1 shows the structure of a single lens reflex electronic camera (hereafter referred to as an electronic 
camera) in which interchangeable lenses are used. The electronic camera 1 includes a camera main body 2 and a 
variable optical system 3 constituted of amounted interchangeable lens. The variable optical system 3 includes a lens 
4 and an aperture 5 . While the lens 4 is constituted of a group of optical lenses, the figure shows a single representative 
55 lens and the position of the lens 4 is referred to as a main pupil position (hereafter simply referred to as a pupil position) . 
The pupil position is represented by a value determined in confomnance to the lens type and the zoom position of the 
zoom lens. The pupil position may also be affected by the focal length. 

[0017] The camera main body 2 includes a shutter 6, an optical component 7 such as an optical filter or a cover 
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glass and an image-capturing elennent (sensor) 8. The variable optical systenn Scan be detachably mounted at a mount 
unit 9 of the camera main body 2. In addition, the variable optical system 3 transmits optical parameters such as 
information related to the pupil position and information related to the aperture value to a control unit 1 7 (see FIG. 2) 
of the electronic camera 1 via the mount unit 9. The aperture value may be selected within a range of, for instance, 
5 F2.8toF22. 

[001 8] Reference numeral 1 0 indicates foreign matter having become adhered to the surface of the optical component 
7 disposed to the front of the image-capturing element 8. Through testing conducted to evaluate changes occurring in 
the foreign matter shadow appearing in photographic images by altering the aperture value and the pupil position at 
the variable optical system 3, the following two findings were obtained. 

10 

(1) The size of the foreign matter shadow and the light transmittance change in correspondence to the aperture 
value. 

(2) The position foreign matter shadow changes as the pupil position of the lens changes. 

^5 These two findings indicate that even when foreign matter remains at a fixed position, it appears to change in the 
photographic image each time the photographing conditions (the aperture value and the pupil position) set for the lens 
are altered. The adverse effect of foreign matter may be eliminated by adopting the method detailed below in conjunction 
with such a variable optical system. 

[0019] FIG. 2 shows a PC (personal computer) 31 and its peripheral devices in addition to a block diagram of the 
20 electronic camera 1 . The PC 31 , which functions as an image processing apparatus, executes the foreign matter effect 
elimination processing to be detailed later by using image data (image digital data) obtained from the electronic camera 
1. 

[0020] The electronic camera 1 includes the variable optical system 3, the optical component 7, the shutter 6 (not 
shown in FIG. 2), the image-capturing element 8, an analog signal processing unit 12, an A/D conversion unit 13, a 
25 timing control unit 1 4, an image processing unit 1 5, an operation unit 1 6, a control unit 1 7, a memory 1 8, a compression/ 
decompression unit 19, a display image generation unit 20, a monitor 21, a memory card interface unit 22 and an 

external interface unit 23. 

[0021] The image-capturing element 8 captures an image of a subject through the variable optical system 3 and 
outputs image signals (image-capturing signals) corresponding to the captured subject image. The image-capturing 

30 element 8 includes a rectangular image-capturing area constituted with a plurality of pixels and sequentially outputs 
the image signals, which are analog signals each corresponding to the electrical charge stored at one of the pixels, to 
the analog signal processing unit 12 in units of individual pixels. The image-capturing element 8 may be constituted 
of a single plate color CCD. The analog signal processing unit 1 2 includes an internal CDS (correlated double sampling) 
circuit, an internal AGC (auto gain control) circuit and the like and executes a specific type of analog processing on 

35 the image signals input thereto. The A/D conversion unit 1 3 converts the analog signals having undergone the process- 
ing at the analog signal processing unit 1 2 to digital signals. The timing control unit 1 4, which is controlled by the control 
unit 1 7, controls the timing with which various operations are executed by the image-capturing element 8, the analog 
signal processing unit 12, the A/D conversion unit 13 and the image processing unit 15. 

[0022] The memory card interface unit 22 achieves interface with a memory card (a card-type removable memory) 
40 30. The external interface unit 23 achieves interface with an external apparatus such as the PC 31 via a specific type 
of cable or wireless transmission line. The operation unit 16 is equivalent to a shutter release button, a mode selector 
button and the like. At the monitor 21 , various menus, a subject image captured at the image-capturing element 8 and 
an image reproduced by using image data stored in the memory card are displayed. The output of the operation unit 
1 6 is input to the control unit 1 7, whereas the output of the display image generation unit 20 is input to the monitor 21 . 
45 The image processing unit 15 maybe constituted of, for instance, a single chip microprocessor dedicated to image 
processing. 

[0023] The A/D conversion unit 1 3, the image processing unit 1 5, the control unit 1 7, the memory 1 8, the compression/ 
decompression unit 19, the display image generation unit 20, the memory card interface unit 22 and the external 
interface unit 23 are connected with one another via a bus 24. 

50 [0024] A monitor 32, a printer 33 and the like are connected to the PC 31 , and an application program recorded in 
a CD ROM 34 is pre-installed at the PC 31 . In addition, the PC 31 includes a memory card interface unit (not shown) 
that achieves interface with the memory card 30 and an external interface unit (not shown) that achieves interface with 
an external apparatus such as the electronic camera 1 via a specific type of cable or a wireless transmission line, as 
well as a CPU, a memory and a hard disk (not shown). 

55 [0025] As the operator of the electronic camera 1 structured as shown in FIG. 1 selects a photographing mode and 
presses the shutter release button via the operation unit 1 6, the control unit 1 7 implements timing control forthe image- 
capturing element 8, the analog signal processing unit 12 and the A/D conversion unit 13 via the timing control unit 
14. The operator may select either a normal mode to photograph a normal subject or a foreign matter information 
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acquisition mode (a reference innage acquisition mode) to pliotograph a reference image, which is to be detailed later, 
in order to obtain foreign matter information, as the photographing mode. It is to be noted that the normal mode may 
include a plurality of sub-modes corresponding to various types of subjects. 

[0026] The image-capturing element 8 generates image signals which correspond to an optical image formed in the 
5 image-capturing area by the variable optical system 3. These image signals undergo a specific type of analog signal 
processing at the analog signal processing unit 12 and are then output to the A/D conversion unit 13 as analog proc- 
essed image signals. The image signals having undergone the analog processing are digitized at the A/D conversion 
unit 13 which then provides the digitized image signals to the image processing unit 15 as image data. 
[0027] It is assumed that in the electronic camera 1 achieved in the embodiment, the image-capturing element 8 is 
10 constituted of a typical single-plate color image-capturing element having R (red), G (green) and B (blue) color filters 
arranged in a Bayer array and that image data adopting the RGB colorimetric system are provided to the image 
processing unit 15. Each of the pixels constituting the image data holds color information corresponding to one of the 
three color components, R, G and B. While each of the photoelectric conversion elements constituting the image- 
capturing element 8 is referred to as a pixel in this context, each unit of the image data corresponding to a pixel is also 
15 referred to as a pixel. In addition, an explanation is given based upon the concept that an image, too, is constituted of 
a plurality of pixels. 

[0028] The image processing unit 15 executes image processing such as interpolation, gradation conversion and 
edge emphasis on such image data. The image data having undergone image processing then undergo a specific type 
of compression processing at the compression/decompression unit 19 as necessary, and are recorded into the memory 
20 card 30 via the memory card interface unit 22. 

[0029] The image data having undergone the image processing are provided to the PC 31 via the memory card 30. 
The image data may be provided to the PC 31 via the external interface unit 23 and a specific type of cable or a wireless 
transmission line. It is assumed that the image data having undergone the image processing have been interpolated 
with color information corresponding to all three color components R, G and B present at each pixel. 

25 

(Foreign Matter Effect Elimination Processing) 

[0030] Next, the processing executed to eliminate the undesirable effects of foreign matter from individual sets of 
photographic image data is explained. In the embodiment, a method whereby a reference image used to obtain foreign 

30 matter information is photographed and the foreign matter effect is eliminated from a plurality of images obtained under 
varying optical photographing conditions by using the reference image is adopted. However, the reference image is 
not constituted of completely uniform or homogeneous white reference data but instead is obtained by photographing 
a blue sky, an almost uniform wall surface, a gray chart, a solid color paper surface or the like. The reference data 
used for this purpose may contain information indicating peripheral darkening of the lens, the gradation of the subject 

35 and the shading at the image-capturing element and the like. The reference data should be the type of data available 
under conditions in which a photographing operation can be performed easily at a readily accessible location and do 
not need to be completely uniform since they are converted to uniform reference data by using an algorithm on the 
image processing side. 

40 (Operation Executed on the Electronic Camera Side) 

[0031] Before photographing an image, the user selects either photographing mode, i.e., the normal mode or the 
foreign matter information acquisition mode described earlier. The electronic camera 1 alters file information in the 
captured image data to be recorded into the memory card 30 in correspondence to the selected photographing mode. 

45 [0032] FIG. 3 presents a flowchart of the processing executed in the electronic camera 1 during a photographing 
operation. This processing is executed when the user selects either photographing mode as described above. In step 
S1 in FIG. 3, a decision is made as to whether or not an instruction to continue with a photographing operation has 
been issued. The processing in step S1 is executed so as to allow the processing in FIG. 3 to be terminated promptly 
once the user issues an instruction to end the photographing operation, except for during the execution of the photo- 

50 graphing operation or while photographic image data are being recorded. If the user has issued an instruction to con- 
tinue with the photographing operation, i.e., if either of the photographing modes has been selected as described 
earlier, the operation proceeds to step S2. Otherwise, i.e., if the user has not selected either of the photographing 
modes and has issued an instruction to end the photographing operation, the processing in FIG. 3 ends. 
[0033] In step S3, a decision is made as to whether or not the selected photographing mode is the foreign matter 

55 information acquisition mode. If the foreign matter information acquisitionmode is currently selected, the operation 
proceeds to step S6. In step S6, the photographing conditions (the aperture, the image processing details, etc.) under 
which a reference image is to be photographed as explained later, are set in the electronic camera 1 . If the foreign 
matter information acquisition mode is not currently selected, i.e., if the normal mode is currently selected, the operation 
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proceeds to step S3. At this time, the photographing conditions under which a subject is to be photographed in a normal 
manner are set in the electronic camera. The settings for the photographing conditions under which the normal pho- 
tographing operation is to be executed may be pre-selected in correspondence to the type of subject, or the settings 
may be adjusted by the user. 

5 [0034] The processing in steps S3 to S5 explained below is executed in the normal mode. In step S3, a decision is 
made as to whether or not a photographing instruction has been issued. If the user has issued a photographing in- 
struction through an operation of the shutter release button orthe like, the electronic camera 1 executes a photographing 
operation in the next step 84, and then the captured image data are recorded into the memory card 30 shown in FIG. 
2 in step 85. The captured image data recorded in step 85 become a correction target of image correction processing 

10 executed by the PC 30, which is to be explained later, If it is decided in step S3 that no photographing instruction has 
been issued or after executing the processing in step 85, the operation returns to step 81 . 

[0035] A file name is set in the captured image data (correction target image data, nonna! image data) recorded into 

the memory card 30 in step 85. The file name contains an extension indicating the file type, and the portion following 
"." (dot) toward the end of the file name constitutes the extension, Specifically, the file name set forthe correction target 
15 image data is presented as " <xx (name inherent to each file)>.<yy (extension) >". The "xx" portion is constitutedof 
characters or symbols, whereas the "yy" portion is constituted of several alphabetic letters which are predetermined 
in correspondence to the file type. 

[0036] In addition, tag information is attached onto the correction target image data. The tag information is metadata 
(data having written therein information on the base data and used in file information management) containing, for 

20 instance, information indicating the settings selected in the electronic camera 1 at the time of file generation and the 
date/time of the file generation. In other words, the tag information includes information related to the captured image 
The tag information attached to the correction target image data does not affect the base image data. Namely, even if 
the tag infomnation is modified, the correction target image data do not become altered. The image data correspond 
to the individual pixels constituting the image. 

25 [0037] In step 85, "JPG" is set for the extension of the file name and tag information indicating that the photographing 
operation was performed in the normal mode is attached. It is to be noted that the "JPG" set in the extension indicates 
that the image file was compressed by adopting the JPEG (Joint Photographic Experts Group) method, which is a 
widely used image data compression method. Namely, the compression/decompression unit 1 9 shown in FIG, 2 com- 
presses the image data through the JPEG method. It is to be noted that a different extension may be used and that 

30 another method may be adopted to compress image data. As an alternative, the image data do not need to be com- 
pressed. 

[0038] The processing in steps 86 through 89 explained below is executed in the foreignmatter information acqui- 
sition mode. In step 86, the photographing conditions under which the reference image is to be photographed are set. 
In this embodiment, the photographing conditions such as the F value (aperture value) and image processing details 
35 are set as follows. It is to be noted that the specific terms used to refer to the individual settings are not limited to those 
used below, and any terms may be used as long as they refer to similar setting contents. In addition, other settings 
may be used as long as foreign matter information can be obtained through the processing to be explained later by 
using the photographic reference image data. 

40 • ISO sensitivity:200 

• Edge emphasis: no 

• Gradation correction: Standard 

• Image quality mode: fine 

• Image size: L 

45 • F value: minimum aperture 

• Exposure correction: no 

[0039] After setting the photographing conditions in step 86, a decision is made in step 87 as to whether or not a 
photographing instruction has been issued. Forthis photographing operation, the user uses a uniform or homogeneous 

50 plane as the subject to obtain a reference image as explained earlier. In response to a photographing instruction issued 
by the user by operating the shutter release button or the like, the electronic camera 1 executes a photographing 
operation in the following step 88, and then, the captured image data (reference image data) are recorded into the 
memory card 30 shown In FIG. 2 in step 89. If it Is decided in step S7 that a photographing instruction has not been 
issued, or after executing the processing in step 89, the operation returns to step 81 . 

55 [0040] As in the case of the correction target image data recorded in step 85, a file name is assigned and tag infor- 
mation is attached to the reference image data recorded into the memory card 30 in step 89. In step 89, "JPQ" is set 
as the extension of the file name and tag information indicating that the photographing operation was executed in the 
foreign matter information acquisition mode is attached. It is to be noted that the extension "JPQ" indicates that the 
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image file was photographed in the foreign nnatter infornnation acquisition mode and was compressed through the 
JPEG method. It is also to be noted that another extension may be set as long as it is different from the extension set 
in step S5 and that the image may be compressed through another method. The image data do not even have to be 
compressed. 

5 [0041] As explained above, the electronic camera 1 varies the file extension depending upon whether the captured 
image data recorded into the memory card 30 were photographed in the foreign matter information acquisition mode 
or the normal mode (step S5, step S9). In addition, the tag information indicating the specific photographing mode in 
which the photographing operation was executed is attached to the corresponding photographic image data file (step 
S5, step S9). Thus, it becomes possible to distinguish the photographic image data recorded into the memory card 30 

10 as either reference image data or correction target image data. 

[0042] The following is an explanation of the photographing procedure which is executed to first photograph a ref- 
erence image in the foreign matter information acquisition mode and then photograph a correction target image in the 
normal mode through the processing explained above, given in reference to FIG. 4. 

15 1) A photographing operation 201 is executed to photograph a uniformplane with the pupil position set to PO and 

the aperture value set to AO, and reference image data 0 are output. 

2) A normal photographing operation 202 is executed with the pupil position set to P1 and the aperture value set 

to A1 , and correction target image data 1 are output. 

3) A normal photographing operation 203 is executed with the pupil position set to P2 and the aperture value set 
20 to A2, and correction target image data 2 are output. 

4) A normal photographing operation 204 is executed with the pupil position set to P3 and the aperture value set 
to A3, and correction target image data 3 are output. Namely, a photographing operation of the uniform plane is 
executed (uniform plane photographing) by turning the electronic camera 1 toward the sky or a wail surface, and 
then a photographing operation is executed as desired by turning the electronic camera 1 toward a subject to be 

25 photographed (normal photographing). It is to be notedthatacorrection target image that is an image photographed 

in the normal mode may be instead referred to as a normal Image, a photographic Image or a viewing image. 

[0043] During this procedure, the aperture value is set to AO so as to photograph a reference image with the aperture 
in the most constricted state within the range over which the aperture value can be adjusted at the variable optical 
30 system 3. The aperture value corresponding to the most constricted state may be, for instance, approximately F22 in 
the case of a standard lens. A correction target image, on the other hand, is photographed by setting the aperture 
value equal to that for the reference image or to a value further toward the open side. 

[0044] The uniform plane photographing operation does not need to be executed repeatedly as long as the state of 
the foreign matter adhered to the optical component remains unchanged. While it is more desirable to execute the 

35 uniform plane photographing operation as often as possible, even foreign matter data obtained once a day prove useful 
under normal circumstances. It is up to the photographer to decide when to execute the uniform plane photographing 
operation. However, if a significant length of time has elapsed since the previous uniform photographing operation, the 
reference data obtained through the previous uniformplane photographing operation may not be reliable enough. Ac- 
cordingly, the reference image data obtained through uniform plane photographing operation may be used only if the 

40 time interval between the uniform plane photographed in operation and the ensuing normal photographing operation 
Is within a specific limit. In addition, the unifomn plane photographing operation does not need to precede a nonrial 
photographing operation. Reference image data obtained through a uniform plane photographing operation executed 
after a normal photographing operation may be used. If a plurality of uniform plane photographing operations have 
been executed before and after the normal photographing operation, the reference image data obtained through the 

45 uniform plane photographing operation executed at a time closest to the normal photographing operation maybe used. 
If, on the other hand, the user is concerned with the possibility of new foreign matter having become adhered to an 
optical component, reference image data selected from the two sets of reference image data corresponding to the two 
uniform plane photographing operations closest in time to the normal photographing operation may be used. 

50 (Operation Executed on the Image Processing Apparatus Side) 

[0045] The PC 31 , which functions as the image processing apparatus, takes in the reference image data and the 
correction target image data described above via the memory card 30 and executes processing for eliminating the 
undesirable effect of the foreign matter. When taking in image data, the PC 31 distinguishes reference image data from 
55 correction target image data so as to be able to automatically select data having been identified as the reference image 
data to be used in the foreign matter effect elimination processing. FIG. 5 presents a flowchart of the processing 
executed at the PC 31 when taking into image data from the memory card 30. In step S11, one of the image files 
recorded in the memory card 30 is selected. 
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[0046] In the following step S1 2, a decision is nnade as to whether or not the image file selected in step S1 1 contains 
reference image data. This decision is made by checking the extension set in step S5 or S9 in FIG. 3, Alternatively, 
the decision may be made by using the tag information attached in step S5 or S9. If the extension "JPG" is set or if 
the tag information indicating that the image was photographed in the normal mode is attached to the image file, it is 

5 judged that the image file does not contain reference image data, i.e., that the image file contains correction target 
image data, and the operation proceeds to step S13. If, on the other hand, the extension "JPQ" is set or the tag 
information indicating that the image was photographed in the foreign matter information acquisition mode is attached 
to the imagef ile, it is decided that the image file contains reference image data and the operation proceeds to step S1 4. 
[0047] In step S13, the image file having been determined in step S12 to contain a correction target image is taken 

10 in from the memory card 30 and is stored into a memory or a hard disk. This image file is handled as a foreign matter 
effect elimination target during the foreign matter effect elimination processing to be detailed later. 
[0048] In step S14, the image file having been determined in step S12 to contain reference image data is taken in 
from the memory card 30 and is stored into the memory or the hard disk. At this time, the image file is stored at a 
location different from the memory area into which the correction target image file is stored in step S13 and with a 

15 different directory address. This image file is used later to obtain a transmittance map when eliminating the undesirable 
effect of the foreign matter, as detailed below. 

[0049] After executing the processing in step S13 or S14, a decision is made in step S15 as to whether or not all the 
image files stored in the memory card 30 have been selected. If there are image files that have not been selected yet, 
the operation returns to step S11 to select one of the files that have not been taken in yet. The processing in step S11 

20 to step S15 described above is repeatedly executed until all the image files are taken in. 

[0050] As described above, when taking reference image data and correction target image data into the PC 31 , each 
image file is determined to contain reference image data or correction target image data based upon the file extension 
or the tag information (step S12), and reference image data and correction target image data are stored into memory 
areas and directory locations different from each other (step S13, step S14). Thus, reference image data can be au- 

25 tomatically selected to be used in the foreign matter effect elimination processing explained below without necessitating 
a user image selection. 

[0051] It is to be noted that while the image data being taken into the PC 31 are distinguished as either reference 
image data or correction target data and are stored into a specific location in correspondence to the data type in the 
explanation given above, the present invention is not limited to this example. For instance, all the image data taken 

30 into the PC 31 may be stored at the same location and reference image data may be selected from the image data 
having been taken into the PC 31 by checking the file extensions and the tag infomnation as necessary. 
[0052] Processing is executed as explained below to eliminate the undesirable effect of foreign matter from the 
correction target image data by using the reference image data and the correction target image data having been taken 
in as described above. It is to be noted that if a plurality of sets of reference image data have been taken in, the 

35 reference image data most suitable for different sets of correction target image data are individually selected in com- 
pliance with the selection criteria explained earlier, i.e., based upon file generation dates/times recorded in the tag 
information attached to the files. 

[0053] It is to be noted that the following explanation is given by assuming that each set of reference image data or 

correction target image data input to the PC 31 functioning as the image processing apparatus include embedded data 
40 which enable the identification of the pupil position and the aperture value. The pupil position data may be obtained 
through a calculation executed based upon recorded data embedded in the photographic data and indicating the lens 
type, the zoom position and the focal point position by using a conversion table. FIG. 6 presents a flowchart of the 
processing executed by the PC 31 . 

45 (Processing on Reference Image Data) 

1) Generation of brightness plane 

[0054] In step S21 in FIG. 6, a brightness or luminance plane is generated. A brightness signal is generated from 
50 the R, G and B signals in correspondence to each pixel [i,j] constituting the reference image data by using the following 
expression (1). [i,j] indicates the position of the specific pixel. 

Y[ij] = (R[iJ] + 2*G[i,j] + B[i,j])/4 (1) 

55 

[0055] Although an analysis can be executed in correspondence to the R, G and B planes individually, the effect of 

a foreign matter shadow basically manifests simply as a signal attenuation regardless of the color component. Accord- 
ingly, the color component signals are converted to a brightness component with which the effect of random noise can 
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be lowered by making the most of all the available information, This also speeds up the processing since only a single 
plane representing the brightness component, instead of the three planes corresponding to R, G and B, needs to be 
analyzed. The RGB ratio for the brightness component generation does not need to be that used in the expression 
above, and instead, it may be set to; R:G:B: = 0.3:0.6:0.1 , for instance. 

2) Transmittance map generation (gain map extraction) 

[0056] In step S22, a transmittance map is generated (a gain map is extracted) by executing thefollowing processing. 



10 2-1) Local standardization processing (gain extraction processing) 



[0057] The reference image data may not be necessarily completely uniform, as explained earlier. For this reason, 
the brightness plane that has been generated may not be completely uniform either. A transmittance signal T[i,j] is 
calculated with the following expression (2) for each pixel by executing standardization (or normalization) processing 

15 to locally standardize each pixel value on such a brightness plane. Namely, the relative ratio of the value corresponding 
to each target pixel [i,j] and the average pixel value over a local range containing the target pixel is calculated. Through 
this processing, any non-uniformity attributable to the gradation, shading or the like in the uniform plane data Is suc- 
cessfully eliminated by using an algorithm and it becomes possible to extract lowered transmittance attributable to the 
foreign matter shadow alone. The transmittance ascertained as described above over the entire image plane is referred 

20 to as a transmittance map (gain map). A transmittance map provides information on defects present In the reference 
Image. It is to be noted that the temn "pixel value" refers to a value indicated by a color signal (color information) 
corresponding to a color component or a value Indicated by the brightness signal (brightness Information) at each pixel. 
The pixel value Is a value within a range of 0 ~ 255 when, for instance, expressed in 1 byte. 



25 



30 



T[U] = 



(2) 



/(2«4-l)(2A + l) 



[0058] The range over which the local average among (2a+1 ) x (2b+1 ) pixels is calculated should be larger than the 
diameter of the foreign matter. Ideally, an area that is at least 3 times as large as the foreign matter shadow should be 
set for the range to obtain accurate transmittance data, "a" indicates the number of pixels present to the left/right of 

35 the target pixel [z,j] and b indicates the number of pixels present above/below the target pixel [i,j]. For instance, if the 
pixel pitch at the image-capturing element 8 is 12jim and the distance between the image-capturing surface and the 
surface on which foreign matter is adhered is 1 .5 mm, the diameter of the large foreign matter ranges over approximately 
1 5 pixels with the aperture value set at F22 and ranges over approximately 40 pixels with the aperture value set at F4. 
Accordingly, "a" and "b" should each be set to 40 so as to set the local averaging range over 81 x 81 pixels. However, 

40 this is only an example and a range containing a different number of pixels may be used. 

[0059] The aperture value greatly affects the seriousness of a foreign matter shadow, and while the effect of small 
foreign matter can be easily eliminated by opening the aperture, the effect of large foreign matter may become less 
noticeable but still be present over a significant area even when the aperture is adjusted to the open side. Depending 
upon the pixel pitch width at the image-capturing element, a round foreign matter shadow may manifest over several 

45 tens of pixels even when the aperture is adjusted toward the open side, and in such a case, the local average must 
be taken over a very large range to result In an Increase In the length of processing time. Under such circumstances, 
the processing may be speeded up by calculating the local average with sub-sampled pixels. 
[0060] The processing for calculating the relative ratio over the (2a+1) x (2b+1) pixel range is referred to as local 
standardization processing (gain extraction processing). The filter used when calculating the relative ratio over the 

50 (2a+1) X (2b+1) pixel range may instead be referred to as a gain extraction kernel. FIGS. 7A and 7B illustrate the local 
standardization processing executed on the brightness plane. FIG. 7A shows the brightness signals at pixels ranging 
along the horizontal direction within the brightness plane. Reference numerals 41 and 42 each indicate an brightness 
signal attenuated due to the presence of foreign matter. FIG. 7B shows the results of the local standardization process- 
ing described above executed on the brightness signals In FIG. 7A. Namely, It shows the results of standardization 

55 processing executed on pixel values within the local range. Reference numerals 43 and 44, which correspond to ref- 
erence numerals 41 and 42 in FIG. 7A, each Indicate the transmittance at a point at which foreign matter Is present. 
As illustrated in FIGS. 7A and 7B, any unevenness in the gradation, shading and like contained in the uniform plane 
data is eliminated, and only the reduction in the transmittance attributable to the foreign matter shadow can be extracted. 
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As a result, the position of the foreign matter and the level of the transnnittance can be ascertained at once. 
2-2) Low pass processing of the transnnittance nnap 

5 [0061] While low pass processing of the transmittance map may be omitted, it is still desirable to execute the low 
pass processing since it proves effective in most cases. The transmittance signal T[i j] contains random noise attrib- 
utable to quantum fluctuations in the brightness signal, and for this reason, if the threshold value decision-making 
described in 2-4) below is executed in an area over which the transmittance levels remain close to 1 and subtle effects 
of a foreign matter shadow are discernible, the foreign matter shadow may be extracted in a mottled pattern due to 

10 randomness. The appearance of the image area can be somewhat improved by grouping the foreign matter shadow 
fragments with a low pass filter, the function of which is expressed in the following expression (3). 

T[ij] = {4*T[ij] 
+2*(T[i-1 j]+T[i+IJ]-fT[i,j-1]+T[i,K1]) 
+1 *(T[i-l,j-l]+T[i-1 ]+T[i+l,j-l)+T[i+l,j+l])}/1 6 (3) 

20 2-3) Statistical analysis of the transmittance map 

[0062] A statistical analysis of the transmittance map is executed to obtain an average value m by using expression 
(4) and determine a standard deviations by using expression (5) over the entire image plane corresponding to the 
transmittance map obtained through the local standardization processing. It is to be noted that Nx and Ny respectively 
25 indicate the total numbers of pixels set along the x direction and the y direction. 



40 

2-4) Threshold value decision-making 

[0063] Basically, the areal ratio of foreign matter signals in the transmittance map is extremely small, and for this 
reason, the level of the random noise (shot-noise) attributable to quantum fluctuations in each transmittance signal is 

45 evaluated through the statistical analysis detailed in 2-3). The presence of such subtle random noise is indicated in an 
enlargement 46 of the area assigned with reference numeral 45 in FIG. 7A. A histogram of the transmittance map 
indicates a normal distribution with the standard deviation a around the average value m (m is a value very close to 
1). FIG. 8 shows the histogram of the transmittance map. Since the transmittance can be assumed to remain unaffected 
by the presence of foreign matter shadows over this fluctuation range, the transmittance can be set to 1 forcibly. In 

50 other words, the threshold value decision-making may be executed in accordance with the following conditions (6) and 
(7), 

if |T[i, j]-m| ^30" then T[i,j]=l . . • (6) 

else T[i, j]=T[i, j] . . . (1) 

[0064] The normally distributed random data account for 99. 7% overthe± 3a range and thus, the effect of the random 
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noise can be substantially elinninated with accuracy. Any signal indicating a transmittance outside the + 3a range can 
be considered to be an abnormal signal which can hardly be explained as attributable to a statistical error and can be 
assumed to represent a phenomenon induced by a lowered transmittance due to the presence of a foreign matter 
shadow. When an abnormality is induced by foreign matter, the abnonnal signal indicates a value smaller than 1 under 

5 normal circumstances, 

[0065] However, some abnormal signals may indicate values larger than 1 although such an occurrence is relatively 
rare. This phenomenon is not attributable to a foreign matter shadow but is attributable to, for instance, an interference 
fringe that manifests as a defect occurring at stria (non-uniformity in the refractive index) at an optical low pass filter 
or the like, increases or decreases the intensity of the incident light. Thus, the method may be adopted to detect a 
10 defect in an optical member as well as to detect foreign matter present in the optical path. In addition, the method may 
be adopted to determine the extent of the effect of a pixel defect manifesting within the image-capturing element. While 
foreign matter present close to the image-capturing element 8 tends to show clearly in a photographic image, even 
foreign matter on the photographic lens which becomes fairly blurred in the photographic image can be identified with 
a high degree of accuracy. 

15 [0066] It is to be noted that threshold value decision-making should be executed in accordance with conditions (8), 
(9) and (1 0) below in order to eliminate the effect of foreign matter shadow alone. 

if IT [i, j ] -ml ^3 CT then T[i,j]=l ...{8) 

20 

else if T[i,j]>l T[i,j]=l ...(9) 

else T[i, j]«T[io] , , . (10) 

25 

[0067] Since the average m used in the decision-making is always a value close to 1 , it may be substituted by 1 . 
[0068] As explained above, two types of defect information, i. e. , map information representing the position of a 
defective pixel Qudged to be defective or defect free by deciding whether or not T = 1) and transmittance information 
indicating the extent of the defect, are obtained at once. It is to be noted that the transmittance map, which indicates 

30 the local relative gains, may be referred to as a gain map instead. 

[0069] Under normal circumstances, defects such as foreign matter are detected by using an edge detection differ- 
ential filter. However, the contrast between the shadows of foreign matter present in the optical path which become 
optically blurred, and their surroundings is extremely low. In such a case, the differential filter, the sensitivity level of 
which is very low, can hardly detect any foreign matter. Instead, the statistical characteristics of the transmittance may 

35 be used in the decision-making processing as described above to detect foreign matter with an extremely high level 
of sensitivity, and as a result, it becomes possible to correct the undesirable effect of foreign matter present in the 
optical path thereby achieving the object of the present invention. 

3) Pupil position conversion of the transmittance map 

40 

[0070] In step S23, the transmittance map is converted corresponding to the pupil position. The pupil position con- 
version is executed when the pupil position set to photograph the reference image and the pupil position set to pho- 
tograph the correction target image are different from each other to convert a foreign matter position in the reference 
image to a foreign matter position at which the foreign matter is predicted to appear as it is viewed from the pupil 
45 position of the correction target image. FIGS. 9A and 9B show how the position of the foreign matter shadow changes 
as the pupil position changes. FIG. 9A shows the relationship among the pupil position, the foreign matter and the 
image-capturing surface of the image-capturing element 8. FIG. 9B shows how the foreign matter shadow moves on 
the image-capturing surface as the pupil position changes. 

[0071] As FIGS. 9A and 9B clearly indicate, the position of the foreign matter appearing within the image becomes 
50 offset along the radius vector from an optical axis 51 , i.e., the image center, as the pupil position changes. Bearing this 
in mind, an offset quantity Ar by which the foreign matter present at a position distance from the optical axis 51 within 
the image by r becomes offset along the radius vector is estimated. When the PO represents the pupil position set for 
the reference image, PO' represents the pupil position set for the correction target image and the foreign matter is 
adhered at a position distanced from the image-capturing surface by 1, Ar be calculated by using expression (11) 
55 presented below. 
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(11) 



It is to be noted that the distance I is a value obtained by converting tlie thicl^ness of the optical component to the 
optical path length in the air. 

[0072] The transmittance mapT[i j] of the reference innage is displaced to [r',G] on polar coordinates [r,0] by using the 
following expression (12) and thus, is converted to a transmittance map T[i,j] on coordinates [ij]. 



The offset quantity Ar increases as the distance from the optical axis 51 increases. At the peripheral area of an actual 
image, the offset quantity may be equivalentto several tens of pixels, depending upon the value set for the pupil position. 

4) F value conversion of the transmittance map 

[0073] In step S24, the transmittance map undergoes an F value conversion. Namely, if the aperture value set to 
capture the reference image and the aperture value set to capture the correction target image are different from each 
other, the foreign matter diameter and the corresponding transmittance in the reference image undergo an F value 
conversion to be converted to a foreign matter diameter and a transmittance corresponding to the aperture value set 
further toward the open side to capture the correction target image. FIGS. 1 0A and 1 0B show how the size of a foreign 
matter shadow changes as the F value representing the aperture value changes. FIG. 10A shows the foreign matter 
shadow at a large F value setting, whereas FIG. 1 CB shows the foreign matter shadow at a small F value setting. As 
FIGS. 10A and 10B clearly indicate, the correlation between the distance 1 from the image-capturing surface to the 
foreign matter position and the foreign matter size T can be expressed as in (13) below by incorporating the defining 
expression of the F value (F = focal length/lens effective aperture). 



[0074] The foreign matter diameter can be expressed as a number of pixels by dividing I by the pixel pitch a(mm / 
pixel) at the image-capturing element. Thus, the foreign matter as a point image is estimated to spread over the width 
r when the aperture is set at F. 

[0075] At the same time, since the foreign matter shadow is considered to be enlarged as light is uniformly irradiated 

on the foreign matter forming the point image from the individual angles of incidence at the lens opened to the aperture 
value, the distribution function of the point image can be assumed to be a function with a completely homogeneous 
spread. Accordingly, by executing homogeneous low pass filter processing expressed with a number of pixels corre- 
sponding to the filter width r , an F value conversion with which the foreign matter diameter and the transmittance can 
be predicted with a high degree of accuracy is enabled. While the low pass filter used in this application is nonnally a 
non-separation type filter formed in a circular shape having a diameter r, a separation type filter formed in a square 
shape having longitudinal and lateral sides set to r may instead be used to speed up the processing. 
[0076] Let us now consider an example in which 1 = 0.5mm, a = 5[im/pixels and a transmittance map corresponding 
to the aperture value F22 is converted to transmittance maps corresponding to F16, F11, F8, F5.6 and F4. The one- 
dimensional filter coefficients of the square separation type filter can be set as indicated in FIG. 1 1 . The low pass filter 
processing is executed both along the vertical direction and along the horizontal direction by using the one-dimensional 
filter coefficients indicated in FIG. 11 . It is to be noted that the one-dimensional filter coefficients corresponding to the 
aperture F16 include a total of seven coefficients with the coefficient 0.5 set at both ends, so as to filter a shadow 
spread over a width represented by an even number over a range that expands uniformly along the horizontal direction 
and the vertical direction with a width represented by an odd number around the target pixel. FIG. 12 shows the aperture 
F1 6 filter represented as a two-dimensional filter. 

[0077] Through the conversion processing described above, the transmittance map of the reference image is con- 
verted to the transmittance map corresponding to the pupil position and the F value set to capture the correction target 
image. In other words, the transmittance map of the reference image is converted to a transmittance map equivalent 
to a transmittance map that would generated under the optical conditions with which the correction target image was 





(13) 
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photographed. 

(Processing on the Correction Target Innage) 
5 (5) Gain correction 

[0078] In step S25 in FiG. 6, a gain correction is executed by using the transmittance map obtained through the 
conversion processing explained above. The gain correction is executed by muitipiying the R, G and B values in the 
correction target image data with the reciprocal of the value indicated by the transmittance signal having undergone 
10 the pupil position / F value conversion, as indicated in expressions (14), (15) and (16) respectively. 

R[iJ]=R[ij]/T'[iJ] (14) 
G[ij] = G[iJ]Ar'[iJ] (15) 



B[iJ] = B[iJ]/T"[iJ] (16) 

20 

[0079] FIG. 1 3 shows how the transmittance is converted through the F value conversion over an area where foreign 
matter of medium size is present. The pixel position is indicated along the horizontal axis, and the transmittance is 
indicated aiong the vertical axis. 

[0080] As described above, once a reference image is photographed with the aperture value set to the smallest value 
25 in a variable optical system, another reference image does not need to be photographed under different optical con- 
ditions Namely, by converting the foreign matter data in the single reference image, an effective correction can be 
achieved. As a result, the onus placed on the user of the electronic camera is greatly reduced. In addition, a very high 
level of sensitivity in the foreign matter detection performance can be maintained without having tophotograph a com- 
pletely uniform image. 

30 [0081] In the electronic camera 1 described above, the extension and the tag information stored in the memory card 
30 together with a reference image data file are different from those stored in the memory card 30 together with a 
correction target image data file. Thus, when the image processing apparatus executes the processing for eliminating 
the foreign matter effect by taking in a data file, reference image data are automatically selected without requiring a 
user data file selection. 

35 [0082] It is to be noted that data files of reference image data and correction target image data appended with ex- 
tensions and tag information contents different from each other are stored in the memory card 30 and the data files 
are then tal<en into the PC 31 via the memory card 30 in the embodiment described above. Instead, such data files 
containing reference image data and correction target image data may be taken into the PC 31 via the external inter- 
faces 23. 

40 [0083] In addition, a transmittance map is generated by executing the local standardization processing and the like 
on a reference image of a supposedly substantially uniform plane photographed by the photographer in the embodi- 
ment. However, there may be in a small pattern or the like present in the reference image considered to be substantially 
uniform by the photographer. This problem can be basically solved by photographing the subject in an unfocused state. 
For instance, the photographer may photograph the subject by positioning it at a position closer than the shortest 

45 photographing distance of the lens. As long as the small pattern is photographed as an unfocused image that changes 
gently over a larger range than the range of the gain extraction kernel corresponding to (2a+1) x)2b+1) pixels, a 
reference image which is uniform enough to fulfill the purpose can be obtained. 

[0084] while the image-capturing element in the embodiment adopts a Bayer array R, G and B colorimetric system, 
it goes without saying that no restrictions whatsoever are imposed with regard to the arrangement of the color filters 
50 as long as the required interpolation processing can be executed. Furthermore, the image-capturing element may 
adopt a different colorimetric system (such as the complementary colorimetric system). 

[0085] Moreover, while an explanation is given above in reference to the embodiment on an example in which the 
present invention is adopted in a single lens reflex electronic still camera in which interchangeable lenses are used, 
the present invention is not limited to this example. The present invention may be adopted, for instance, in a camera 
55 that does not allow the use of interchangeable lenses. The pupil position and the aperture value can be ascertained 
by using an appropriate method in the known art. 

[0086] In addition, while image data obtained by photographing images with the electronic still camera 1 are proc- 
essed in the embodiment explained above, the present invention is not limited to this example. The present invention 
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may be adopted to process image data photographed with a video camera which handles dynamic images as well. It 
can also be adopted to process image data photographed with a portable telephone equipped with a camera. The 
present Invention may be further adopted In conjunction with photocopiers, scanners and the like. In other words, the 
present invention may be adopted to process all types of image data obtained by capturing images with an image- 

5 capturing element, 

[0087] While an explanation Is given above In reference to the embodiment on an example In which the effect of 
foreign matter Is eliminated at the PC (personal computer) 31 by processing image data photographed with the elec- 
tronic camera 1, the present invention Is not limited to this example. A program enabling such processing may be 
installed in the electronic camera 1. The program may instead be installed in a printer, a projection apparatus or the 

10 like. In other words, the present invention maybe adopted in all types of the apparatuses that handle image data. 

[0088] The program executed by the PC 31 may be provided in a recording medium such as a CD ROM or through 
a data signal on the Internet or the like. FIG. 14 shows how the program may be provided In those forms. The program 
is provided to the PC 31 via the CD ROM 34. In addition, the PC 31 can be connected with a communication line 101 . 
A computer 1 02 is a server computer that provides the program stored In a recording medium such as a hard disk 103. 

15 The communication line 1 01 may be a communication line for Internet communication, personal computer communi- 
cation or the like, or it may be a dedicated communication line. The computer 1 02 reads out the program from the hard 
disk 103 and transmits the program to the PC 31 via the he communication line 101 . In other words, the program is 
provided as a data signal on a carrier wave transmitted via the he communication line 1 01 . Thus, the program can be 
distributed as a computer-readable computer program product adopting any of various modes such as a recording 

20 medium or a carrier wave. 

[0089] The above described embodiments are examples, and various modifications can be made without departing 
from the spirit and scope of the invention. 



25 Claims 

1. An Image-capturing apparatus, comprising: 

an image-capturing unit that captures an Image of a subject through an optical system; and 
30 a control unit that outputs images captured at the Image-capturing unit as image data files, wherein: 

the control unit outputs the Image data files in a format that allows an Image data file containing a normal 
image to be distinguished froman image data file containing a reference image to be used to correct an 
image defect in the normal Image. 

35 

2. An Image-capturing apparatus according to claim 1 , further comprising: 

a mode setting unit that sets a reference image acquisitionmode in which the reference Image Is to be obtained, 
wherein: 

40 

the control unit designates an Image captured by the image-capturing unit when the reference image 
acquisition mode is set as the reference image. 

3. An Image-capturing apparatus according to any one of claims 1 through 2, wherein: 

45 

the control unit appends different extensions to the image data file containing the nomnal image and the image 
data file containing the reference image. 

4. An Image-capturing apparatus according to any one of claims 1 through 3, wherein: 

50 

the Image data files are each constituted of image data corresponding to pixels in the image and information 

related to the image; and 

the control unit stores Information with which the Image data file containing the normal Image and the image 
data file containing the reference image can be distinguished from each other in the infomnatlon related to the 
55 image. 

5. An Image-capturing apparatus comprising: 
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an image-capturing unittliat captures a subject image input tlirough an optical system; 
a mode switching unit tliat selects a first mode in which a reference image is captured by the image-capturing 
unit or a second mode in which a normal image is captured by the image-capturing unit; 
an output unit that outputs the reference image and the normal image captured by the image-capturing unit 
5 as image data files to an external apparatus; and 

an information appending unit that appends information indicating whether an image data file output by the 
output unit contains the reference image or the normal image to the image data file. 

6. An image-capturing apparatus according to claim 5, wherein: 

10 

the information appending unit appends metadata containing information indicating eitherthe reference image 
or the normal image to the image data file; and 

the meta data do not alter the reference image or the normal image contained in the image data file. 

^5 7. An image-capturing apparatus according to any one of claims 5 through 6, wherein: 

the information appending unit appends the information indicating whether the image data file contains the 
reference image or the normal image to the image data file by partially modifying a file name assigned to the 
image data file. 

20 

8. An image processing apparatus, comprising: 

an input unit to which an image data file output from an image-capturing apparatus according to any one of 
claims 1 through 4 is input; 

25 an identifying unit that identifies the image data file as a file containing the reference image or a file containing 

the normal image; and 

a correction unit that corrects an image defect in the image identified as the normal image based upon the 
image identified as the reference image. 

30 9. An image processing apparatus according to claim 8, wherein: 

the image data files are each constituted of image data corresponding to pixels in the image and information 
related to the image; and 

the correction unit selects a specific reference image to be used to correct an image defect in a given nomnal 
35 image based upon the information related to the image contained in the image data file of the reference image 

and the information related to the image contained in the image data file of the normal image. 

10. An image processing apparatus, comprising: 

40 an input unit to which an image data file output from an image-capturing apparatus according to any one of 

claims 5 through 7 is input; 

an identifying unit that identifies the image data file as a file containing the reference image or a file containing 
the normal image based upon the information; and 

an elimination unit that eliminates an effect of foreign matter present on an image-capturing optical path at the 
45 image-capturing unit from the normal image based upon the reference image. 

1 1 . A computer-readable computer program product comprising: 

a control program that executes a function of an image processing apparatus according to any one of claims 
50 8 through 10. 



55 



15 



EP1 478 169 A2 



FIG.1 




16 



EP1 478 169 A2 



o 

Ql 



o 
tr 
I 

o 
o 



1 



i 





BUS 



UN 



ANALOG 
PROCESS 



lO 



VD COWERS ION 



IT 



C3 



o 



SIGNAL 
NG UNIT 



IMAGE-GAPTURING 
ELEMENT 



T7 



LLICS 

— CO 
LU 

O 

o 




o 



\m CONTROL 
UNIT 



OPTICAL COMPONENT 



VARIABLE OPTICAL 
SYSTEM 




17 



EP1 478 169 A2 



FIG.3 




S4 


Y 


S8 


Y 


PHOTOGRAPH 




PHOTOGRAPH 


S5 


S9 




RECORD CORRECTION TARGET 
IMAGE DATA 




RECORD REFEREr 


ICE IMAGE DATA 



18 



EP1 478 169 A2 



5 



I— <c 



Q_ 



1 



O 

cvi 



D- 
—I CD 



PC 

ex. 

-J CD 



_ a. < _ 

J csr 2 

1 CL- 



•<: 

UJ 



I — 

Q- 



E i 

oL err 



^ or 



Oil 

<t:o 



o 
cvi 



UJCD 

lac 

Qc: CD 
<3>c:> 

— o 



qI 



O 



a. 



o 



CO 

o 



a. 



or 

UJ 



o 



Lur 



UJ 

2e 



19 



EP1 478 169 A2 



FIG.5 



C 



START 



1 



S11 



SELECT IMAGE FILE 



SI 2.,^ MAGE PHOTOGRAPHED l!r\ Y 
FOREIGN MATTER INFORMATION 
ACQUISITION MODE? 



TAKE IN AS NORMAL IMAGE 



N 




SI 4 



TAKE IN AS FOREIGN MATTER 
INFORMATION IMAGE 



END ^ 



20 



EP1 478 169 A2 



F1G.6 



( CORRECTION 
I TARGET IMAGE J 



PUPIL POSITION PC 
APERTURE VALUE AO' 




PUPIL POSITION PO 
APERTURE VALUE AO 



8ENERATE 
BRIGHTNESS PUNE 



S21 



S22 



GENERATE TRANSM I TTANCE MAP 
(EXTRACT SAIN MAP) 

1 ) LOCAL STANDARD I ZAT I ON PROCESS I NG 

2) STATISTICAL ANALYSIS 

3) THRESHOLD VALUE DECISION-MAKING 



(^^ITTANCE^^^ 



I 



EXECUTE PUPIL POSITION 
CONVERSION (PO -> PO') 



S25 



EXECUTE GAIN 
CORRECTION 



EXECUT 
CONVERSION 



S23 



F VALUE 
(AO -> AC ) 



S24 



CORRECTED IMAGE 



21 



EP1 478 169 A2 




EP1 478 169 A2 



C3 



o 
I— 



X 



CO 



CO 




if S 



O 



CO 
LL, 



23 



EP1 478 169 A2 



LENS 



FIG.9A 



IMAGE-CAPTURING 
SURFACE 



OPTICAL 
COMPONENT 




• ^ V 1 


i::;:::: 


< 

■ «» 

0 


1 H 

1 N 



FIG.9B 



BEFORE DISPLACEMENT: PUPIL POSITION PO 
AFTER DISPLACEMENT: PUPIL POSITIOHPO' 
WHEN PO > PO' 



24 



EP1 478 169 A2 



LENS 



FIG.10A 



IMAGE-CAPTURING 
SURFACE 



OPTICAL 
COMPONENT 





FIG.10B 



25 



EP1 478 169 A2 



FIG.11 



APERTURE VALUE 


FOREIGN MATTER 
DIAMETER r 
[pixel] 


LPF SIZE 


ONE-DIMENSIONAL FILTER 
COEFFICIENTS 


F22 


4.5 


DOES NOT REQUIRE CONVERSION PROCESSING 


F16 


6.3 


6x6 


[0. 5,1.1,1.1,1.0.53/6 


Fll 


9.1 


9x9 


[1,1.1,1,1,1.1.1,13/9 


F8 


12. 5 


13x13 


[1, 1. 1. 1. 1, 13/13 


F5. 6 


17. 9 


18x18 


[0. 5, 1, 1. t. 1. 0. 51/18 


F4 


25.0 


25x25 


[1, 1, 1 1. 1, 13/25 



26 



EP1 478 169 A2 



FIG.12 



0.25 


0.5 


0.5 


0.5 


0.5 


0.5 


0.25 


0.5 


1 










0.5 


0.5 


1 










0.5 


0.5 


1 










0.5 


0.5 


1 










0.5 


0.5 


1 










0.5 


0.25 


0.5 


0.5 


0.5 


0.5 


0.5 


0.25 



736 



EP1 478 169 A2 



FIG. 13 



F VALUE CONVERSION FOR TRANSMinANCE MAP 
(SECTIONAL VIEW) 




30 40 $0 

Pixel Position 





F22 (original) 


o 


F16 (IJ>F) 


♦ 


FIT (IPf) 


o 


F8 OJ>f) 


T 


F5.6 (LPF) 




F4 (LPF) 



60 



-T — ] — 1 — I — r— 
70 



28 



EP1 478 169 A2 



FIG.14 




29 



